A Real-World Look at AWS Best Practices: Logging

Introduction

Best practices for securing an AWS environment have been well-documented and generally accepted, such as AWS’s guidance. However, organizations may still find it challenging on how to begin applying this guidance to their specific environments.

• Which controls should be applied out-of-the-box vs. customized?
• What pitfalls exist in implementing the various controls or checks?
• How do you prioritize remediation of the “sea of red” violations?

In this blog series, we’ll analyze anonymized data from Netskope customers that include security settings of 650,000 entities from 1,143 AWS accounts across several hundred organizations. We’ll look at the configuration from the perspective of the best practices, see what’s commonly occurring in the real world and:

• Discuss specific risk areas that should be prioritized
• Identify underlying root causes and potential pitfalls
• Focus on practical guidance for applying the Benchmark to your specific environment

This blog post focuses on IAM security controls related to logging. Based on the Netskope dataset analyzed, we will highlight four opportunities to improve security:

1. Enable VPC flow logs: 81% of VPCs do not have VPC flow logging enabled, which will hinder incident response and investigations.

2. Encrypt CloudTrail logs at rest: 91% of CloudTrail logs are not encrypted at rest. Encryption at rest supports data compliance controls and is easy to do.

3. Ensure S3 bucket access logging is enabled for CloudTrail buckets: 41% of CloudTrail buckets do not have server access logging enabled. Logging should be enabled for all CloudTrail S3 buckets.

4. Ensure CloudTrail logs are integrated with CloudWatch or a SIEM: 54% of CloudTrails are not integrated with CloudWatch. These should be reviewed to ensure they are integrated with a production log search service or SIEM.

Logging

These nine technical best practices involve logging configuration, including AWS CloudTrail, bucket access logging, and VPC flow logs. They were analyzed against 1,485 CloudTrails, 11,101 VPCs, and 16,281 keys across 1,143 accounts in the Netskope customer dataset.

1. CloudTrail enabled in all regions

Background: CloudTrail should be enabled for all regions to ensure complete activity logging is available, including activity in unused regions is detected.

Data: In this dataset, 39 accounts (3.4%) do not have a CloudTrail enabled for all regions.

Analysis: This control is crucial as it is a prerequisite for visibility into having logging enabled in the first place in all regions. Any violations should be immediately remediated.

Controls: 

• Detection/Audit
  • CloudTrail should be enabled for all regions and regular, automated checks on CloudTrail configurations should be done to enforce this setting.
  • The CloudTrail configuration setting, IsMultiReegionTrail, can be checked via CLI:
    
    aws cloudtrail describe-trails
    
    
  • AWS Config has a rule that will also detect if CloudTrail is not enabled for all regions.

• Prevention/Mitigation
  • The CloudTrail setting, IsMultiRegionTrail, can be configured via CLI:
    
    aws cloudtrail update-trail
    

2. CloudTrail log file validation

Background: A digitally-signed hash of each CloudTrail log can be created on the same S3 bucket where the CloudTrail logs are stored. This is invaluable in ensuring the integrity of the logs and determining if a log was changed or deleted after CloudTrail wrote the log. Enable file validation on all logs to provide additional integrity checking of CloudTrail logs.

Data: 271 out of 1,485 CloudTrail logs (18%) do not have file validation enabled.

Analysis: Log file validation is easy to turn on and should be enabled for all CloudTrail logs.

Controls: 

• Detection/Audit
  • The CloudTrail log file validation setting, LogFileValidationEnabled, can be checked via CLI:
    
    aws cloudtrail describe-trails
    
    
  • AWS Config has a rule that will also detect if log file validation is not enabled.

• Prevention/Mitigation
  • To set the CloudTrail setting, LogFileValidationEnabled, use the CLI:
    
    aws cloudtrail update-trail


  • One can also validate log files for a specified period of time, on demand with this CLI command:
    
    aws cloudtrail validate-logs

3. CloudTrail private bucket

Background: CloudTrail logs are stored in an S3 bucket, which should not be publicly accessible.

Data: In this dataset, two out of 1,485 CloudTrail buckets are public (0.13%).

Analysis: There is only a very small percentage of CloudTrail buckets that are public. However, remediating the two public buckets is critical as the CloudTrail logs are at risk of being readable or possibly writable publicly, which leaks sensitive information and puts log integrity at risk.

Controls: 

• Detection/Audit
  • The CloudTrail bucket’s public settings can be checked via CLI:
    
    aws s3api get-public-access-block

aws s3api get-bucket-policy

aws s3api get-bucket-acl
    
    
  • AWS Config also has 3 rules that can detect buckets with public access:
    • s3-bucket-level-public-access-prohibited
    • s3-bucket-public-read-prohibited
    • s3-bucket-public-write-prohibited

• Prevention/Mitigation
  • To mitigate a public CloudTrail bucket, apply the Public Access Block, which will ensure that the bucket is not only private but will prevent it from accidentally being changed to public by blocking modification of the bucket policies or ACLs.
  • The Console and CLI provide easy ways to set the Public Access Block setting:
    
    aws s3api put-public-access-block

4. CloudTrail integrated with CloudWatch

Background: To provide better searching, metrics, and monitoring alerts, CloudTrail should be integrated with CloudWatch or a product such as a SIEM.

Data: 799 out 1,485 CloudTrails (53.8%) are not integrated with CloudWatch. 

Analysis: About half of the CloudTrails are not integrated with CloudWatch. If CloudWatch is the standard service used for metrics and alarms, then these CloudTrails should be remediated immediately. If an alternative product, such a SIEM, is the standard, then there are mitigating controls, and this control can be deprioritized.

Controls: 

• Detection/Audit
  • CloudWatch configuration for CloudTrails can be checked via CLI:
    
    aws cloudtrail describe-trails
    
    
  • AWS Config has a rule to check for CloudTrail configuration with CloudWatch.

• Prevention/Mitigation
  • To configure CloudTrail to integrate with CloudWatch log groups, use the CLI:
    
    aws cloudtrail update-trail

5. AWS Config enabled in all regions

Background: AWS Config can detect common misconfigurations and also is the basis of AWS compliance findings in AWS SecurityHub. It should be enabled in all regions.

Data: 104 out of 1,143 accounts (9.1%) do not have AWS Config enabled for all regions.

Analysis: AWS Config should be enabled for all regions in every account.

Controls: 

• Detection/Audit
  • Ensuring that AWS Config is enabled involved 3 checks which can be done via CLI:
    
    aws configservice describe-delivery-channels

aws configservice describe-configuration-recorders

aws configservice \ describe-configuration-recorder-status

• Prevention/Mitigation
  • To turn on AWS Config with the CLI:
    
    aws configservice put-configuration-recorder

aws configservice put-delivery-channel

6. CloudTrail bucket has access logging enabled

Background: S3 access logging should be enabled on the CloudTrail bucket to ensure an audit trail exists for all accesses to this sensitive bucket.

Data: 610 out of 1,485 CloudTrail buckets (41.1%) do not have S3 server access logging enabled. 

Analysis: Without server access logging, unexpected access to or modification of the CloudTrail logging bucket may not be detected. This should be remediated immediately.

Controls: 

• Detection/Audit
  • The logging status of a bucket can be checked via CLI:
    
    aws s3api get-bucket-logging
    
    
  • AWS Config will also check whether S3 bucket logging is enabled.

• Prevention/Mitigation
  • To enable bucket logging, use the CLI:
    
    aws s3api put-bucket-logging

7. CloudTrail logs encrypted at rest

Background: It is recommended to encrypt CloudTrail logs at rest by utilizing the AWS KMS service and customer-managed keys (CMKs). This supports data privacy compliance. Several encryption options are available and are free, often with little work required.

Data: 1,358 out of 1,485 CloudTrails do not have encrypted logs (91.4%).

Analysis: Encryption at rest is not popular, and at a minimum should be used to support data privacy compliance efforts.

Controls: 

• Detection/Audit
  • A bucket’s encryption policy can be checked via CLI:
    
    aws s3api get-bucket-encryption
    
    
  • AWS Config can also specifically check that the CloudTrail bucket has server-side encryption using an AWS KMS CMK.

• Prevention/Mitigation
  • To set bucket encryption, use the CLI:
    
    aws s3api put-bucket-encryption

8. Customer CMKs are rotated

Background: The encryption keys used to encrypt CloudTrail logs should be rotated regularly to mitigate key compromise. AWS provides a configuration setting that will automatically rotate KMS CMKs.

Data: Overall, 3,338 out of 16.281 keys (20.5%) are not rotated automatically.

Analysis: To provide better security for encryption best practices, encryption keys need to be rotated. With one-fifth of all keys not rotated, encryption practices are weakened with the use of old keys. Enabling automatic key rotation is easy to do and improves security.

Controls: 

• Detection/Audit
  • The KMS rotation policy can be checked via CLI:
    
    aws kms get-key-rotation-status
    
    
  • AWS Config has a rule to check whether KMS automatic key rotation is enabled.

• Prevention/Mitigation
  • To set automatic key rotation, use the CLI:
    
    aws kms enable-key-rotation

9. VPC flow logging is enabled for all VPCs

Background: VPC flow logging should be enabled for packet rejects for all VPCs. Should an incident occur, this supports investigations into associated suspicious traffic.

Data: 9,015 out of 11,101 VPCs (81.2%) do not have flow logging of any kind enabled.

Analysis: Most VPCs do not have VPC flow logging enabled. Flow logging is instrumental to network traffic investigations and this should be prioritized for remediation.

Controls: 

• Detection/Audit
  • VPC flow logging can be checked can be done via CLI:
    
    aws ec2 describe-flow-logs
    
    
  • AWS Config has a rule to check if a VPC has flow logging enabled.

• Prevention/Mitigation
  • VPC flow logging can be enabled with the CLI:
    
    aws ec2 create-flow-logs

Conclusion

Many best practices have been codified but many AWS environments lag behind in implementing these best practices. Remediating the issues is straightforward for many of the security settings, and there exists specific prescriptive guidance on auditing and remediating your configurations in these areas, which can result in a large reduction in risk.

Here are some basic measures that can be done to address some of the common risk areas due to logging in your AWS environment:

1. Enable VPC flow logs: Flow logs can be easily enabled and support incident response and investigations. They should be treated with the same priority as API logging with CloudTrail.

2. Encrypt CloudTrail logs at rest: Encrypt CloudTrail logs to protect log integrity. Encryption at rest supports data compliance controls and is easy to do.

3. Ensure S3 bucket access logging is enabled for CloudTrail buckets: CloudTrail logging is crucial and to ensure integrity of the logs, S3 access logging should be enabled for all CloudTrail buckets.

4. Ensure CloudTrail logs are integrated with CloudWatch or a SIEM: CloudTrail logs are best searched or processed in CloudWatch or a SIEM. Any CloudTrails that do not have an associated CloudWatch log group should be reviewed to ensure they are integrated with a production log search service or SIEM.

In upcoming blogs, we’ll explore other best practices and how individual organizations can apply these best practices specific to their environment.

Additionally, Netskope’s Public Cloud Security platform also can automate configuration checking of your AWS environment, implementing both compliance standards, as well as custom configuration checks.

Dataset and Methodology

Time Period: Data was sampled/analyzed from January 24, 2021. 

Source: The analysis presented in this blog post is based on anonymized usage data collected by the Netskope Security Cloud platform relating to a subset of Netskope customers with prior authorization.

Data Scope: The data included 1143 AWS accounts and several hundred organizations. 

The data was composed of configuration settings across tens of thousands of AWS entities including IAM users, IAM policies, password policy, buckets, databases, CloudTrail logs, compute instances, and security groups.

Logic: The analysis followed the logic of core root account security checks found in best practices regarding AWS configuration settings with a few adjustments for the dataset and methodology. Some best practices might define “recent usage” for the root account as a last logged-in time occurring within the past 24 hours to determine whether the root account has been used recently. Because this dataset comes from a point-in-time snapshot, this was changed to within the past 7 days prior to the audit date.